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SYSTEM AND METHOD FOR CAPTURING COLOR IMAGES THAT EXTENDS 
THE DYNAMIC RANGE OF AN IMAGE SENSOR 



5 FIELD OF THE INVENTION 



The invention relates generally to digital imaging systems, and more 
particularly to digital imaging systems that are adapted to capture images with 
high contrast between bright and dark areas. 

10 



BACKGROUND OF THE INVENTION 



Color digital imaging systems, such as digital cameras, typically employ a 

15 single image sensor, either a charged coupled device (CCD) or a complementary 
metal oxide semiconductor (CMOS) device, to digitally capture a scene of 
interest. Image sensors typically include an array of tightly spaced optical 
detectors, such as photodiodes, that generate an electrical response in 
proportion to the intensity of incident light. The dynamic range of the individual 

20 optical detectors is defined by the minimum amount of light that is required to 

generate an electrical response on the low end and the maximum amount of light 
beyond which the electrical response of the optical detector does not change (the 
saturation point) on the high end. 

The dynamic range of an image sensor is an important characteristic 

25 when capturing high contrast images. When the bright and/or dark areas of an 
image exceed the dynamic range of the image sensor, the quality of the captured 
image suffers. If the sensitivity of the image sensor is adjusted (i.e., by 
decreasing the exposure time) to sufficiently capture the features of bright areas 
in an image, then the features of the dark areas are not captured. Likewise, if the 

30 sensitivity of the image sensor is adjusted (i.e., by increasing the exposure time) 
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to sufficiently capture the features of dark areas in an image, then the features of 
the bright areas are not sufficiently captured. 

One known technique for capturing high contrast scenes with a digital 
image sensor involves capturing two images of the same scene in rapid 

5 succession, with the sensitivity of the image sensor set to capture the bright 

areas in a first image and the dark areas in a second image. The two images are 
then used to generate a composite image that includes features in both the bright 
and dark areas. Fig. 1 depicts an example of a two-image technique in which the 
features of bright areas (i.e., background scenery) are captured in a first image 

10 1 02 using a first exposure time and the features of dark areas (i.e., a person in 
the foreground) are captured in a second image 104 using a second exposure 
time. Features of the bright areas from the first image and features of the dark 
areas from the second image are then used to generate a combined image 106 
that includes features of both the bright and dark areas. 

15 Although the two-image technique can extend the dynamic range of an 

image sensor, the technique requires enough memory to store two images 
before the combined image is generated and enough processing power to 
generate the combined image. In addition to the memory and processing 
requirements of the two-image technique, any changes in the scene that occur 

20 between the capturing of the first and second images will degrade the quality of 
the combined image. 

In view of the problems involved with capturing high contrast color images 
with digital imaging systems, what is needed is a system and method for 
extending the dynamic range of digital imaging systems. 

25 

SUMMARY OF THE INVENTION 

A method and system for extending the dynamic range of a digital imaging 
30 system involves simultaneously capturing image data from bright and dark areas 
of a high contrast image using an image sensor with pixels that have different 
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sensitivities to the bright and dark areas. The image data from the different 
sensitivity pixels is then used to generate a final image that includes features 
from both the bright and dark areas. Because the technique captures all of the 
image data simultaneously with one image sensor, the technique requires half 

5 the time and half the memory of conventional two-image techniques. Because 
there is no need to merge multiple images into a final image, the technique 
requires less processing power and time than the two-image techniques. 
Additionally, because all of the image data is captured simultaneously, the 
temporal consistency of the image data is maintained even if the scene includes 

10 significant motion. 

An embodiment of a digital imaging system includes an image sensor with 
a first group of pixels that have a first sensitivity to a first color and a second 
group of pixels that have a second sensitivity to the same color. Image data that 
is captured by the two groups of pixels is brought to a common scale before the 

15 image data is used to generate demosaiced image data. The scaled image data 
is used in a demosaicing process to determine intensity values for the first color 
at all of the pixel locations. Errors in the demosaiced intensity values that result 
from capturing the first color image data with pixels of two different sensitivities 
are corrected to generate a final image that accurately depicts the original high 

20 contrast color image. 

A method for capturing a color image with a digital imaging system 
involves capturing color-specific image data with a first group of pixels that have 
a first light sensitivity and simultaneously capturing color-specific image data with 
a second group of pixels that have a second light sensitivity, wherein the color- 

25 specific image data from the first group of pixels and the color-specific image 

data from the second group of pixels are for a first color and wherein the first and 
second light sensitivities include a common dynamic range. In additional steps, 
the image data from the first group of pixels and the image data from the second 
group of pixels is brought to a common scale, intensity values of the first color 

30 are determined for pixels using the scaled color-specific image data, and 



Attorney Docket 10003979-1 



4 



determined intensity values that are outside the common dynamic range of the 
first and second groups of pixels are corrected. 

Other aspects and advantages of the present invention will become 
apparent from the following detailed description, taken in conjunction with the 
5 accompanying drawings, illustrating by way of example the principles of the 
invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

10 

Fig. 1 depicts a prior art two-image technique in which the details of a first 
image and the details of a second image are used to generate a final image. 

Fig. 2 depicts a digital imaging system that is used to capture color digital 
images. 

15 Fig. 3 depicts an array of color-specific pixels that includes a distribution of 

green pixels with two different sensitivities in accordance with an embodiment of 
the invention. 

Fig. 4 depicts example outputs from a Gi pixel and a G 2 pixel relative to 
their minimum and maximum outputs when the pixels are subjected to relatively 
20 bright light. 

Fig. 5 is an expanded view of an embodiment of the image processing unit 
of Fig. 2 in accordance with an embodiment of the invention. 

Fig. 6 depicts the relative scales of the output values of the R, d, G 2 , and 
B pixels shown in Fig. 3 in the case where the pixels have 10-bit output fields and 
25 the ratio of the sensitivities of the Gi and G 2 pixels is 1 :4 (G1/G2 = V*). 

Fig. 7 depicts the scaled 12-bit output fields of the R, d, G 2 , and B pixels 
shown if Fig. 6. 

Fig. 8 depicts an array of color-specific pixels that can be used to calculate 
a demosaiced green intensity value for the red pixel. 
30 Fig. 9 depicts an example error correction scheme for a demosaiced 

green intensity value that exceeds the top dynamic range of the d pixels. 
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Fig. 10 depicts a process flow diagram of a method for capturing a color 
image with a digital imaging system in accordance with an embodiment of the 
invention. 

5 

DETAILED DESCRIPTION OF THE INVENTION 

Fig. 2 depicts a digital imaging system 208 that is used to capture color 
digital images. The digital imaging system includes a shutter 210, imaging optics 
10 212, an image sensor 214, an image processing unit 216, memory 218, and a 
display 220. In an embodiment, the digital imaging system is a hand-held digital 
camera that is used to capture still digital images. In another embodiment, the 
digital imaging system is a digital video camera that is used to capture digital 
video. 

15 The shutter 210 and imaging optics 212 are conventional devices that are 

well known in digital imaging systems. Likewise, the memory 218 and display 
220 are conventional devices that are well known in digital imaging systems. In 
an embodiment, the shutter, imaging optics, memory, and display are integrated 
with the image sensor 214 and the image processing unit 216 into a digital 

20 camera. The image sensor and image processing unit are the focus of the 
invention and are described in more detail below. 

The image sensor 214 is an array of individual optical detectors, referred 
to as pixels, which individually detect light and generate electrical responses in 
proportion to the amount of incident light. Common optical detectors include 

25 CCDs and CMOS devices. Both CCD and CMOS devices are only sensitive to 
illumination. Consequently, these optical detectors cannot discriminate between 
different colors. In order to achieve color discrimination, a color filtering 
technique can be applied to separate light in terms of primary colors, typically 
red, green, and blue. A common filtering technique utilizes a color-filter array 

30 (CFA), which is overlaid on the image sensor, to separate colors of incident light 
in a Bayer pattern. A Bayer pattern is a periodic pattern with a period of two 
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different color pixels in each dimension (vertical and horizontal). In the horizontal 
direction, a single period includes either a green pixel and a red pixel, or a blue 
pixel and a green pixel. In the vertical direction, a single period includes either a 
green pixel and a blue pixel, or a red pixel and a green pixel. In an image sensor 

5 that utilizes the Bayer pattern, the number of green pixels is twice the number of 
red or blue pixels. The reason for the disparity in the number of green pixels is 
that the human eye is not equally sensitive to red, green, and blue. 
Consequently, more green pixels are needed to create a color image of a scene 
that will be perceived as a "true color" image. 

10 Fig. 3 depicts an array of color-specific pixels 314 that includes a 

distribution of green pixels (Gi and G 2 ), with two different sensitivities. In the 
embodiment of Fig. 3, a Bayer pattern pixel distribution is used such that the 
number of green pixels is twice the number of red or blue pixels. In the 
horizontal direction, one period includes a blue pixel (B) and a green pixel with a 

15 first sensitivity (Gi) and the other period includes a green pixel with a second 

sensitivity (G 2 ) and a red pixel (R). In the vertical direction, one period includes a 
blue pixel (B) and a green pixel with the second sensitivity (G 2 ) and the other 
period includes a green pixel with the first sensitivity (GO and a red (R) pixel. 
Given the above described green pixel distribution, over a large surface area of 

20 the image sensor, the total number of Gi pixels is approximately equal to the 
total number of G 2 pixels. The ratio of Gi pixels to G 2 pixels is an important 
factor in the error correction process described below. Although the color 
spectrum of the image sensor is divided into red, green, and blue, in other 
embodiments, the color spectrum can be divided differently. 

25 In the embodiment of Fig. 3, the Gi pixels are more sensitive to the dark 

(or shadow) areas of a scene and the G 2 pixels are more sensitive to the bright 
(or highlight) areas of a scene. That is, the pixels are better adapted to 
distinguish the features in dark areas of a scene while the G 2 pixels are better 
adapted to distinguish the features in bright areas of a scene. In an embodiment, 

30 the Gi pixels are four times as sensitive as the G 2 pixels, such that the sensitivity 
ratio of the d pixels to the G 2 pixels is 1 :4. For example, when the Gi and G 2 
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pixels are exposed to the same intensity of light for the same period of time, the 
G 2 pixels generate output values that are four times higher than the output values 
of the Gi pixels. 

In an embodiment, the sensitivity of the pixels is determined by amplifier 

5 circuits within the image sensor. In adjustable image sensors, the gain of the 
pixel amplifier circuits can be adjusted through control signals that are sent to the 
image sensor. Preferably, the amplifier gain for the different color pixels is set at 
the center of the dynamic range of the particular color that is targeted by the 
pixels. For example, the amplifier gain of the red pixels is set to the expected 

10 center of the red dynamic range. In an embodiment, the gain of the Gi and G 2 
pixels can be adjusted to account for the expected level of contrast in the image 
that is to be captured. The adjusted gain of the G 1 and G 2 pixels must be 
reflected in the scales of the Gi and G 2 output fields. Scaling of the pixel output 
fields is described in more detail below. 

15 Fig. 4 depicts example outputs from a Gi pixel and a G 2 pixel relative to 

their minimum and maximum outputs when the pixels are subjected to the same 
amount of relatively bright light. As shown in Fig. 4, the bright light exceeds the 
top end of the G-\ pixel dynamic range and therefore the output of the Gi pixel is 
a value that does not accurately reflect the intensity of the incident light. 

20 Specifically, the output of the Gi pixel is limited to the maximum value possible in 
the dynamic range of the Gi pixel. For example, if the pixel has a 10-bit output 
field, the output value of the Gi pixel in the bright conditions is 1,023. In contrast, 
the bright light does not exceed the dynamic range of the G 2 pixel and therefore 
the output of the G 2 pixel is a valid output that accurately reflects the intensity of 

25 the incident bright light. Because the Gi pixels produce inaccurate outputs in 
bright conditions, errors in the image data generated from the optical sensor of 
Fig. 3 should be corrected in the final image. A technique for correcting the 
errors is described below with reference to Figs. 9 and 10. 

Referring back to Fig. 2, image data generated from the image sensor 214 

30 is forwarded to the image processing unit 216. The image processing unit 

receives image data from the image sensor and uses the image data to generate 
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a complete set of image data that can be used to display a final image. Fig. 5 is 
an expanded view of an embodiment of the image processing unit of Fig. 2. In 
the embodiment of Fig. 5, the image processing unit 516 includes a scaling unit 
524, a demosaic unit 526, an error correction unit 528, and a processor 530. 

5 The scaling unit 524 scales the outputs from the red, green, and blue 

pixels so that all of the output values are on a common scale. Fig. 6 depicts the 
relative dynamic ranges of the R, Gi, G 2 , and B pixels on a log 2 scale in the case 
where 10-bit output fields have been scaled to a common 12-bit output field. Fig. 
7 depicts the 12-bit scaled output fields of the R, d, G 2 , and B pixels relative to 

10 each other. With respect to the Gi and G 2 output values, Fig. 6 depicts the 

relative scales of the output values of the Gi and G 2 pixels, in the case where the 
ratio of the sensitivities of the Gi and G 2 pixels is 1:4 (Gi/G 2 = %). That is, when 
the Gi and G 2 pixels are exposed to the same intensity of light for the same 
period of time, the G 2 pixel output values are four times higher than the Gi pixel 

15 output values. Given the sensitivity ratio of 1 :4, the output values of the Gi and 
G 2 pixels can be brought to a common scale by adding two bits (both zero) to the 
top end of the d output field and by adding two bits (both zero) to the bottom 
end of the G 2 output field, such that the 12-bit output field of the Gi pixels is: 
d: 00 XX XX XX XX XX, 

20 and the 1 2-bit output field of the G 2 pixels is: 
G 2 : XX XX XX XX XX 00. 

As shown in Fig. 6, the scaled Gi pixels cover a range of values from 0 - 
1 ,023 and the scaled G 2 pixels cover a range of values from 0 - 4,092. When 
the pixels are used in combination, the total dynamic range covered by the 

25 combination of the Gi and G 2 pixels includes values 0 - 4,092. That is, the 1 0-bit 
output fields of the Gi and G 2 pixels are synthetically made to appear as 12-bit 
output fields when the Gi and G 2 pixels are used in combination on the same 
image sensor. Although the total dynamic range of the Gi and G 2 pixels is 0 - 
4,092, the top end of the common dynamic range is set by the Gi pixels at a 

30 maximum value of 1 ,023, which is the maximum value of the 1 2-bit output field 
with the two most significant bits set to zero. The red and blue pixels (R and B) 
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are brought to the same scale as the green pixels by adding two bits to the 10-bit 
output fields to make 12-bit output fields. In the embodiment shown in Figs. 6 
and 7, the red and blue pixels are brought to the common scale by adding one bit 
(a zero bit) to the top end of the R and B output fields and by adding one bit (a 

5 zero bit) to the bottom end of the R and B output fields. Given this scaling, the 
12-bit output fields of the R and B pixels are: 
R and G: OX XX XX XX XX XO. 

The relative resolution of the R, Gi, G 2 , and B pixels is indicated by the 
density of the hatching in the bars of Fig. 6. Specifically, the resolution of the d 

10 pixels is the highest (as indicated by the relatively dense hatching), the resolution 
of the G 2 pixels is the lowest (as indicated by the relatively wide hatching), and 
the resolution of the R and B pixels is between the Gi and G 2 pixels. 

Referring back to Fig. 5, once all of the pixel values are brought to a 
common scale, the pixel values for the R, Gi, G 2 , and B pixels are forwarded to 

15 the demosaic unit 526 as a mosaiced image. The demosaic unit operates to 
demosaic the input mosaiced image such that each pixel of the resulting 
demosaiced image has an intensity value for all three primary colors (red, green, 
and blue). That is, each pixel of the demosaiced image should have an intensity 
value for red, green, and blue. The demosaic unit estimates the missing intensity 

20 values for each pixel of the input mosaiced image using available intensity values 
from surrounding pixels of the same color. There are many different 
demosaicing techniques and algorithms that can be used in conjunction with the 
image sensor of Fig. 3 to generate a demosaiced image. One basic demosaicing 
technique involves calculating an intensity value for a particular color by finding 

25 the average intensity value of nearby pixels of the same color. Referring to Fig. 8 
for example, a green intensity value for a red pixel location can be calculated as 
the average green pixel value for the immediately surrounding Gi and G 2 pixels. 
That is, the green intensity value for the red pixel shown in Fig. 8 is calculated as 
Gred pixel location = (Gi + G 2 + + G 2 )/4. Although the calculation described with 

30 reference to Fig. 8 takes only four green pixel values into consideration, typical 
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demosaicing techniques use a larger set of pixel values to determine an intensity 
value. 

The output of the demosaic unit is a demosaiced image that includes red, 
green, and blue intensity values for each pixel location. As described above with 

5 reference to Fig. 4, the green Gi pixels are more sensitive to dark areas and will 
give range-limited output values (i.e., 1,023) for bright areas that exceed the top 
end of the Gi pixel dynamic range. The range-limited output values will cause 
the demosaiced green intensity values that exceed the top end of the common 
dynamic range to be artificially low. Because some of the demosaiced intensity 

10 values related to bright areas include artificially low values, the image that is 

output from the demosaic unit may still lack the full dynamic range that is covered 
by the G-i and G 2 pixels. 

The error correction unit 628 corrects the intensity value errors that result 
from using a distribution of pixels with two different sensitivities. Referring to Fig. 

15 6, if the demosaiced green intensity value of a pixel exceeds the top end of the 
common dynamic range of the Gi and G 2 pixels (i.e., point A), then it can be 
assumed that light incident on at least some of the Gi pixels exceeds the top end 
of the Gi pixel dynamic range. Although the light exceeds the top end of the Gi 
pixel dynamic range, the output of the Gi pixels is range-limited to 1,023 and 

20 therefore the demosaiced green intensity value is held artificially low. Because 
the G 2 pixels are more sensitive to bright areas, it can be assumed that the G 2 
pixels give accurate output values in response to the incident light. 

In an embodiment of an image sensor in which the number of d pixels is 
approximately equal to the number of G 2 pixels per unit of surface area on the 

25 image sensor, the demosaiced green intensity values that exceed the common 
dynamic range of the d and G 2 pixels represent approximately one-half of the 
difference between the accurate G 2 pixel values and the top end of the d 
dynamic range. This is because one-half of the pixel values used for 
demosaicing are the G 2 pixel values (representing valid readings) and the other 

30 one-half of the pixel values used for demosaicing are the d pixel values, which 
are range-limited to a maximum output value of 1 ,023. In order to correct the 
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error in a demosaiced green intensity value, the difference between the 
demosaiced green intensity value and the maximum intensity value of the Gi 
pixels (i.e., the top of the common dynamic range) is doubled to generate the 
error corrected green intensity value. Fig. 9 depicts the error correction for a 

5 demosaiced green intensity value that exceeds the dynamic range of the Gi 
pixels. As shown in Fig. 9, if the demosaiced green intensity value exceeds the 
maximum intensity value of the G^ pixels by X, then the error corrected green 
intensity value is doubled such that the final demosaiced green intensity value 
exceeds the maximum intensity value of the Gi pixels by 2X. Although in the 

10 example of Fig. 9 the error corrected intensity value is determined by a linear 
function, in other embodiments, error corrected intensity values can be 
determined by a non-linear function [f(x)]. 

In an embodiment, the demosaiced green intensity values that are within 
the common dynamic range are not error corrected and the demosaiced green 

15 intensity values that are outside of the common dynamic range are error 
corrected but not beyond the top end of the G 2 dynamic range. 

The processor 530 may perform additional image processing on the 
capture image data. In an embodiment, the processor supports the demosaic 
unit 526 and the error correction unit 528. 

20 Fig. 10 depicts a process flow diagram of a method for capturing a color 

image with a digital imaging system in accordance with an embodiment of the 
invention. At step 1002, color-specific image data is captured with a first group of 
pixels that have a first light sensitivity. At step 1004 color-specific image data is 
simultaneously captured with a second group of pixels that have a second light 

25 sensitivity, wherein the color-specific image data from the first group of pixels and 
the color-specific image data from the second group of pixels are for a first color 
and wherein the first and second light sensitivities include a common dynamic 
range. At step 1006, the image data from the first group of pixels and the image 
data from the second group of pixels are brought to a common scale. At step 

30 1 008, intensity values of the first color are determined for pixels using the color- 
specific image data from the first and second groups of pixels. At step 1010, 
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determined intensity values that are outside the common dynamic range of the 
first and second groups of pixels are corrected. 

Although the process of scaling the red, green, and blue pixel output 
values to a common scale is described as coming before the process of 

5 demosaicing, this is not a requirement. In other embodiments, demosaicing can 
be done before the red, green, and blue pixel output values are brought to a 
common scale. Although the image sensor is described with pixels that have 1 0- 
bit output fields, the number of bits in the output field can be any number of bits. 
For example, 8-bit output fields may be used. Additionally, although a 

10 sensitivity/scaling ratio of 1:4 is described above, other sensitivity/scaling ratios 
may be used. For ease of scaling, ratios that are factors of two are preferred. 
For example, for scenes with extremely high contrast, the d and G 2 pixels are 
scaled as described with reference to Figs. 6 and 7 (i.e., shifting the G 2 pixels by 
two bits), however for scenes with less contrast, the Gi and G 2 pixels can be 

15 scaled by shifting the 1 0-bit field of the G 2 pixels by one bit similar to the R and G 
pixels. 

Although the distribution of green Gi and G 2 pixels in the array of Fig. 3 is 
approximately even, in other embodiments of the array, the distribution of same- 
color pixels with different sensitivities may be uneven. In embodiments of 
20 uneven Gi and G 2 pixel distribution, the function that is used to correct errors in 
the demosaiced green intensity values is adjusted as necessary to account for 
the uneven distribution. Additionally, although the green pixels are described 
with two different sensitivities, other color-specific pixels can be used with 
different sensitivities instead of, or in addition to, the green pixels. 
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